[TransferEngine] Rebuild single Jetty on ACK timeout (status=9) - #33

Defer endpoint deletion until after poll depth accounting to avoid UAF,
deliver flush completions during rebuild, and isolate stale CQEs with
per-slot jetty epochs. Align disconnect teardown with ERROR flush fence
constraints and document the updated rebuild accounting model.

Co-authored-by: Cursor <cursoragent@cursor.com>

Co-authored-by: Cursor <cursoragent@cursor.com>

Extend the mock URMA provider so CI can drive the status=9 rebuild path
without real hardware: record each posted WR's jetty local id, script the
next poll status (URMA_CR_ACK_TIMEOUT_ERR), inject a FLUSH_ERR_DONE fence,
return WR_FLUSH_ERR from urma_flush_jetty, and optionally fail the next
urma_create_jetty. Add gtest cases TC-1..TC-3 covering the happy-path
rebuild, flush-CR delivery, and stale-epoch drop, and wire them into the
tent-ci (ub-mock) job as a scoped ctest.

Co-authored-by: Cursor <cursoragent@cursor.com>

…h test

TC-2 failed on CI because the single-JFC/single-slot fixture drained both
posted WRs in the first poll (s1 failed with status 9, s2 succeeded), leaving
nothing for the rebuild flush loop to deliver. Add a withhold hook to the mock
so a posted WR stays outstanding until urma_flush_jetty completes it, and post
s2 with that hook so the flush path is genuinely exercised.

Co-authored-by: Cursor <cursoragent@cursor.com>

The rebuild flush loop delivers each outstanding WR through
processWrCompletion and increments poll's resolved_wr_count, so delivering s2
via flush yields resolved == 1 (the fence marker itself is not counted). Fix
the test to assert that instead of 0.

Co-authored-by: Cursor <cursoragent@cursor.com>

Rebuild failure (urma_endpoint.cpp, urma_endpoint.h): when urma_delete_jetty
fails mid-rebuild, keep the old jetty handle and mark the slot
REBUILDING_FAILED instead of nulling it, so deconstruct can retry the delete
rather than leak the handle. The new state is never selected for posting.

Drain timeout (urma_endpoint.cpp): when the flush-done fence never arrives,
checkDrainTimeout now flushes the jetty and delivers each residual WR through
processWrCompletion (endpoint still alive) so stuck slices are failed/retried
and depth accounting returns to zero before the endpoint is deferred for
deletion. disconnectUnlocked keeps its drain-only flush (endpoint is being
torn down), with a comment clarifying why CRs are not delivered there.

Add TC-4 (rebuild failure keeps the handle, deferred delete) and TC-5 (drain
timeout delivers residual WRs) to urma_jetty_rebuild_test.

Co-authored-by: Cursor <cursoragent@cursor.com>

TC-4 was driving rebuild failure via fail_next_create_jetty, which fails in
recreateJettyUnlocked after the old jetty was already deleted — so the handle
is legitimately null and the slot stays REBUILDING. The handle-preservation
fix targets urma_delete_jetty failure instead. Add a fail_next_delete_jetty
hook to the mock and point TC-4 at that branch, which is where the old handle
must be kept (REBUILDING_FAILED) for deconstruct to retry.

Co-authored-by: Cursor <cursoragent@cursor.com>